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ABSTRACT 



This paper considers the setting of evaluation from the 
perspective of the situations and stages of problem solving in which 
institutional leaders might call for the assistance of evaluators. This 
discussion should provide evaluators with a framework to understand and 
adjust their interactions with the environmental context of the evaluation. 

R. Heifetz has posited three types of situations for institutional leaders: 

(1) clear problem and clear solution; (2) clear problem and unclear solution; 
and (3) unclear problem and unclear solution. For this discussion, a problem 
solving process is defined that contains five steps, from problem definition 
to making changes. Another aspect of the situation in which institutional 
leaders might or would seek the assistance of evaluators is the definition of 
evaluation whether it determines the extent to which goals have been 
accomplished, provides descriptive information, or judges the merits of a 
program or components. Evaluators have different roles to play in the 
institutional setting of their work, and their roles depend on the situation. 
When there is clarity about the problem and the solution, the work is within 
purposes that are summative or formative, and thus basically technical. When 
there is lack of clarity about the problem or the solution, the work expands 
into sociopolitical and psychological purposes and becomes both technical and 
interactive. (SLD) 
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Clinical Context of Evaluation: Interaction of Evaluation 
with Situation Types and Stages of Problem Solving 

As evaluators we work in a context of people, leaders, stakeholders, situations. Like 
researchers, evaluators should seek to identify the elements of the context in which they work. 

Unlike researchers, who seek to separate from the context in order to gain new, generalizable 
knowledge, evaluators must embrace the context in order to provide the specific 
knowledge/information necessary to meet the needs of the stakeholder client. 

The purpose of this presentation is to examine the setting of evaluation from the perspective of 
the types of situations and the stages of problem solving wherein institutional leaders might be calling 
for the assistance of evaluators. As a result of this examination evaluators would have a framework 
for understanding and adjusting their interactions with the environmental context of the evaluation. 

For the purpose of this discussion, situational types, as defined by Ronald Heifetz in Leadership 
without Easy Answers, will be utilized. In his delineation situation based issues revolve around the 
clarity or lack of clarity of the problem feeing the institutional leader and the clarity or lack of 
solution and implementation procedures for the solution. Heifetz posits three types of situations for 
leaders based on these aspects: 

Type I: the problem is clear and the solution and implementation are also clear (that is, 

neither requires adaptation or learning). The identification of the problem and the 
solution and implementation of the solution are essentially technical operations. 

The leader and those led have seen the problem before and worked out acceptable 
ways for resolving the problem. The leader’s role essentially is to define the 
problem and implement protocols) already developed to resolve the problem. 

Type II: the problem is clear but the solution and implementation are unclear (that is, the 
solution and/or implementation requires adaptation or learning). In this type the 
problem is definable but protocols for resolving the problem have not been worked 
out. The leader may have ideas about the solution but acceptance and 
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implementation of the solution requires that those led — and perhaps the leader as 
well — have to change/adapt in some way. The leader may have a technical fix (that 
is, be able to spell out what ought or can be done to resolve the problem) but those 
being led control acceptance and implementation of the problem solution. 

Type III: the problem is unclear (it requires adaptation or learning in order to identify the 
problem) and the solution and implementation is unclear, also. In Type III 
situations the problem is not clearly definable and when it is defined, protocols for 
fixing it do not exist. Adaptation and learning is required both to define the problem 
and to determine and implement a solution to the problem. (Heifetz, 1994) 
According to Heifetz, administrative leadership behaviors should differ based on the situation. 
Where the problem, solution, and implementation are clear, the appropriate behaviors would include 
providing the problem definition and solution, protecting followers from outside threats, restoring 
order, and maintaining group/institutional norms. Where the problem, solution, and/or 
implementation are unclear, the appropriate behaviors include identifying the learning challenge, 
diagnosing conditions, asking questions about problem definition and solutions, disclosing external 
threats, expose conflicts, and either challenging group/institutional norms or allowing them to be 
challenged. 

Switching now to problem solving processes, in attempting to deal with all types of problem 
situations, administrative leaders have available a variety of problem solving schema. For the sake of 
this discussion a problem solving process with five major steps — and in which the fifth step has four 
substantive sub-steps — is utilized. 

1. Problem Definition — processes that result in clarifying both what the problem is and is 
not with the goal of identifying the problem underlying the symptoms that have raised the 
issue to the attention of the institution. 

2. Team Creation — putting together a team of people who have the knowledge, skills, and 
institutional influence or “reach” to resolve the identified problem. Alternatively, the 
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team might be established prior to the problem’s clear definition; in that case, the team 
must be able both to define and to resolve the problem. 

3. Analysis of Causes — processes that lead to a determination of potential causes for the 
problem with the goal of identifying the root cause of the identified problem. 

4. Solution Identification — processes that identify the available solutions/responses to the 
root cause with the goal of determining the best solution for the current situation. 

5. Plan and Make Changes — processes that begin with the identified best solution and move 
to its institutionalization, assuming, of course, that it resolves the identified problem. 

This stage has four major sub-steps. 

a) Plan the Change — identifying and laying out the tasks that must be done for the 
identified best solution to move from idea to reality. Its product would normally 
be an action plan detailing who does what, when, and with what resources. 

b) Pilot the Change — a process of trying out the proposed solution in a limited, 
somewhat controlled environment for the purpose of determining if it works, how 
well it works, and what adjustments to the plan might be needed before full 
implementation takes place. 

c) Check the Change — evaluation processes to determine how well the pilot worked 
with the goal of determining whether to go forward and institutionalize the 
change, adjust the plan is some significant way(s), or to revisit the solution 
identification stage of the problem solving process. 

d) Institutionalize the Change — processes that move the change from outside the 
systems of the institution to becoming a permanent part of the institutional 
system. 

In addition to situation types and the problem solving process, a third aspect of situations in 
which institutional leaders might/would seek the assistance of evaluation experts lies within the 
definitions and purposes of evaluation. Three typical definitions of evaluation are presented across 



the literature: (1) it is the determination of the extent to which program goals have been 
accomplished; (2) it is the provision of descriptive information to decision makers; and (3) it is 
judging the merit or worth of a program or its components. 

Within each definition, a variety of purposes can motivate the actions of evaluators. The most 
commonly noted purposes are summative and formative evaluations. In summative evaluation the 
purpose is to contribute to decisions about program installation, continuation, or expansion. 

Formative evaluation seeks to contribute to decisions about program modifications. Two additional 
purposes tend to be of importance to institutional leaders: psychological or sociopolitical purposes 
and the exercise of authority. Psychological or sociopolitical purposes revolve around efforts to 
motivate desired behavior by the people in the program being evaluated. It contributes to the 
dynamic tension necessary to produce learning and change in behavior. Obtaining evidence that 
rallies support or opposition for the program is one way this purpose is fulfilled. Lastly, there is the 
need or desire of the institutional leader to exercise his/her authority. Put simply, the evaluation is 
made to happen to demonstrate the leader’s ability to make it happen. (Nevo, 1990) 

When the institutional leader initiates a problem solving process, the leader has need of 
evaluation services that are primarily formative both in the operation of the problem solving process 
and in addressing the problem that was the impetus for invoking the problem solving process. Within 
the problem solving process, evaluators can contribute at each step although the characteristics of the 
contributions change from step to step. 

Problem definition might have begun with the results of an earlier evaluation cycle. In that 
sense the problem solving process is the product of evaluation; it is an action to resolve some 
weakness identified in a prior evaluation event. Beyond that, once the problem solving process is 
begun, evaluation processes can be used to confirm the existence of a gap between reality and 
potential in the identified area. Where a gap is confirmed, evaluation can provide information about 
the size of the gap. This is useful since the existence of a large gap offers the possibility of significant 
benefit from effort and resources applied to closing the gap while a narrow gap might indicate that. 
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although the gap is real, the institution’s time and resources might achieve more cost effective returns 
in other areas. For example, if the results of the sixth grade proficiency tests show 20 students out of 
200 have not passed the math portion and 100 out of 200 have not passed the reading portion, the 
expenditure required to get the last 20 students to pass the math would probably have a greater return 
on effort if it were applied to the 100 students who had not passed the reading portion. 

Also in the problem definition stage, evaluators can help to determine one or more means for 
noting and/or measuring success. Perhaps the same methods that enable the institution to determine 
there is a gap between the reality and potential can be used to identify milestones of progress toward 
complete success. More likely, the evaluator would be able to assist the institutional leader to 
identify a level of success realistic for the resources able to be committed to the solution of the 
problem. In a sense, the evaluator helps the institution to recognize its limits given the current 
situation. This can assist the institutional leader in moving the institution toward a revision of its 
priorities as a means to address the problem in a more substantive way than the status quo would 
allow. 

Team creation offers several ways for evaluators to be of assistance. The evaluator can 
examine the problem, however delineated at this point in the process, and help to identify the human 
resources — knowledge, skills, authority, and influence — necessary both to clearly identify the 
problem and to resolve it. Additionally, resources external to the team can be evaluated both as to 
their availability and their potential for resolving the problem at hand. These resources might be 
external to the team but internal to the institution, or they may be external both to the team and the 
institution. 

Analysis of causes would benefit from the assistance of an evaluator in two ways. First, the 
evaluator could work with the team to identify likely root causes of the problem. That is, the 
evaluator can help the team move beyond the visible, surface symptoms of the problem and track 
back to a cause that, if treated, could eliminate the problem altogether. Normally, this would be 
preferable to attacking the symptoms, reducing or removing them for the immediate present only to 



have them reoccur, probably at the most inopportune time possible. Secondly, the evaluator can help 
the team by developing a means to verify that the suspected root cause actually exists and is the root 
cause of the undesirable symptoms. 

Solution identification follows the determination of root causes. Typically many solutions are 
brainstormed in the problem solving process, and the key is to find the one that best fits the context of 
the situation. Evaluation processes can help the team to define the criteria of the ideal solution and 
then to evaluate the alternative solutions against the identified criteria. In carrying out this work, the 
team and even more, the evaluator must be cognizant of the entire institution as a system. It is 
important to have a sense of the level of change the solution will require of the institution. The 
deeper and more profound the change, the more costly in social capital will be the implementation of 
the solution. That, in turn, requires an understanding of how much change the institution can stand in 
its current state and in a later state, after some level of preparation. 

Planning and making the change involves four major activities: (1) design the change, (2) pilot 
the change, (3) check or evaluate the change, and (4) act or institutionalize the change. 

During the process of designing the change, evaluation can play a role in the identification of 
the major steps that necessary to bring about the change. Forms of process evaluation would be 
useful to the planners. Related to that is generating a process to monitor the planned changes. The 
major steps could well be the progress milestones of the plan, so developing the data collection and 
analysis that would enable the implementers to track progress and adjust the plan would enable the 
institutional leaders to carry out the plan as designed or as adapted to fit changed circumstances. 

Pilot the change has several points where an evaluator can be of assistance. First lies in 
determining whether a pilot implementation would be beneficial or a misuse of time and/or other 
resources. Second is to assist in the decision about where the pilot could be implemented with 
meaningful effect. 

Once the pilot is set into motion, the next set of actions are essentially all evaluative. It is 
necessary to monitor the progress of the pilot and to determine its effectiveness. Evaluation 
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opportunities and needs exist at nearly every point in the pilot process. To what extent does the 
planned pilot program embody the theory or ideas of the full program? To what extent has the pilot 
been implemented as planned? What are the outcomes being produced by the implemented pilot? 
How effective is the pilot at producing the outcomes? How does this relate to the expected 
outcomes? What lessons does the pilot offer toward improving the program and/or its 
implementation? Are the outcomes worth the cost? In short, this stage potentially involves every 
type of evaluation activity. Finally, evaluation assists in making the summative decision, should the 
proposed solution be implemented in full flower? 

At the point the decision is to fully implement or institutionalize the solution, evaluation’s role 
shifts to formation. Now the evaluator helps to identify the training needed in order to bring 
permanence to the solution. This is coupled with the need to identify systemic changes that are 
needed for the solution to be institutionalized. Among the systemic changes would be the 
identification and implementation of checks and controls that verify the solution is and continues to 
be active. Aspects of the solution are noted, counted, or measured when active both to assure that 
they are present and to symbolize the institutional intention to make them permanent. Finally, 
evaluators can help to determine the effectiveness of the institutionalized solution. In a sense, the 
evaluation cycle begins anew. 

To help visualize these relationships, the chart. Evaluator Function and Purpose in Various 
Circumstances, is inserted below. In this chart, the problem solving process moves from the center of 
the chart up to the top (from Problem Identification to Act/Institutionalize). Four purposes of 
evaluation lie below the center of chart. In the center, moving from left to right, are the three 
situation types defined by the clarity of the problem and the solution/implementation. Each type is 
further divided into problem and solution. Where the situation type and the problem solving steps 
intersect, codes have been entered to reflect the evaluator’s role. Where it is primarily one of 
technical assistance where the product of the evaluation is based on evaluation techniques and 
knowledge and the effects of interaction with stakeholders is incidental to the product, “T” is entered. 



Where the effects of interaction with the stakeholders is integral to the product or benefit to the 
adaptive needs of the institution, “I” is entered. The “I” code does not mean technical skills are not 
needed; it means that the product that benefits the institution is drawn from the evaluator’s skills and 
processes in interaction with the stakeholders as well as from technical skills and knowledge. The 
level of technical skills needed might be even higher than where “T” is coded, since they must be 
exposed or practiced in a visible, open setting. Where no major benefit seems present, the cell is left 
blank. 

In the lower part of the chart, the coding is “P” for primary purpose and “S” for secondary 
purpose, and blank where the purpose does not seem needed. The line labeled “Exercise Power” is 
coded with a “U” where showing the decision maker’s authority has potential utility, under certain 
circumstances, in the move to bring about adaptive change in the institution. For example, where the 
problem and/or the solutions are unclear and the stakeholders believe the authority figure is doing 
nothing, authorizing an evaluation could be a means to contain or reduce the tensions of the moment, 
buying time to get processes moving toward clarifying the problem and/or solution. 

Moving now to the consideration of how these three factors intersect affect the role of the 
evaluator. With the Type I situations, both problem and solution clear, the institutional leader, that is, 
the person in authority, the decision maker, is the client for the services of the evaluator. In the phase 
of problem definition, the evaluator’s role would be threefold, (1) to verify and establish the 
parameters of the problem, (2) to determine if the/an existing team has the means to resolve the 
problem, and (3) to determine the potential for external resources to contribute to the resolution of the 
identified problem. In short, the evaluator checks to confirm that the problem is what it is thought to 
be. This can be identified as a formative evaluation function, assisting toward the implementation of 
processes for resolving an essentially unquestioned problem. 

Similarly, the evaluator’s role in the implementation of the solution is essentially formative. A 
solution or a variety of solutions are already known and generally acceptable to the institution. The 
evaluator can engage in four contributory ways: (1) to identify criteria for selecting among existing 
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Evaluator Function and Purpose in Various Circumstances 
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solutions, (2) to identify mileposts and a monitoring process for solution implementation, (3) to verify 
that the implementation is being accomplished as planned, and (4) to verify that the objectives of the 
implemented solution have been met. 

So in Type I situations the evaluator is engaged in primarily formative purposes. He/she uses 
technical skills related to evaluation to confirm the shape the problem to the decision maker and to 
inform the decision maker of the progress of the solution being implemented. There is the additional 
benefit of documenting the process for any accountability purposes that might exist within or without 
the institution, but the work is essentially formative and technical. The effect of the evaluators 
contacts with the stakeholders is immaterial to the adaptive process. 

The Type II situation is a mixed bag. The problem is clear — there is agreement and acceptance 
both that there is a problem and what it is. The solution, on the other hand, is not clear — there is not 
agreement or acceptance about how to resolve the problem. In terms of problem definition. Type II is 
the same as Type I. The person in authority, the decision maker, is the institutional leader and the 
tasks for the evaluator are to verify and establish the parameters of the problem. Slightly different, 
though, are the tasks related to the team and to external resources. Here, with the problem clear and 
verified but the solution not clear, the evaluator’s task in problem definition expands to include 
determining desirable characteristics and/or talents needed on the team assigned the task of 
developing an as yet unknown or unclear solution. Then the evaluator determines if the 
identified/existing team meets those criteria. The evaluator’s task related to external resources would 
be in the same vein, determine desirable characteristics of potential external resources and then 
determining if the presently available external resources meet the criteria. The uncertainty of the 
solution affects the work of the evaluator by reducing the level of certainty in identifying the team 
and external resources, yet the client is still the institutional leader who needs essentially formative 
and technical information. 

To this point the work of the evaluator has been essentially transactional. That is, the 
institutional leader and the stakeholders in the institution came to grips with the type of problem at 
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some earlier time and worked out mutually acceptable ways of disposing of the problem. The 
evaluator has provided essentially technical services to a client that is informed and accepting about 
the situation. The leader sees to the implementation of the anticipated solution. The evaluator assists 
the leader is this, providing information supportive of both the problem identification and the solution 
implementation. The evaluator is not seeking to change either the leader’s or the stakeholders’ view 
of the problem or their mutually acceptable solution. 

On the other hand, with the solution and its implementation unclear, the evaluator becomes part 
of the process to generate new knowledge or new ways to respond to a problem, that is, the a new 
protocol for dealing with this new or unresolved problem. This means the evaluation process has a 
psychological or sociopolitical purpose, that is, to contribute to obtaining the stakeholders’ 
acceptance of the new protocol. In this context there are more actions to be taken to generate a 
solution than when there is an institutionally accepted protocol for resolving this clearly identified 
and accepted problem. Intertwined in the technical aspects of solving the problem, there is the need 
to develop the acceptance or will to implement the newly developed solution. Here the effects of the 
evaluators interactions with the stakeholders plays a role that is likely to be larger than that played by 
the evaluation report(s). 

The evaluator can begin the process with a determination of causes. However, instead of 
independently searching out causes of the identified problem and authoritatively presenting them, the 
evaluator helps the institutional leader in the adaptive change process by involving stakeholders in 
processes that enable them to establish both the complexity of causes and the root causes. The 
evaluator educates them to go beyond treating the symptoms. 

With causes and root causes in hand, the evaluator can work with the stakeholders to develop 
criteria for the resolution of the problem. There may be a need to work through criteria for the ideal 
solution and for a solution that all can live with before settling on the criteria that will be applied to 
the evaluation of the potential solutions once they are generated. Once the criteria are established, the 
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evaluator can assist the stakeholders by gathering information about the various solutions proposed by 
the stakeholders or by others external to the institution. 

After a solution is identified for the problem, the evaluator can assist the stakeholders by 
helping them identify the major milestones of an action plan to implement the solution. This would 
go hand in hand with developing a plan to monitor the implementation of the plan. If the plan 
requires a major investment in time, money, or personnel to fully implement, the evaluator can assist 
the stakeholders work through a pilot process, providing information about where the pilot could be 
meaningfully implemented and monitor the progress and effectiveness of the pilot. This information 
could also be developed and presented to the stakeholders so that they can decide if the piloted 
change is ready for institutionalization. 

If the stakeholders decide the proposed solution is not ready for full implementation, the 
evaluator helps them identify what needs improving before full implementation is judged acceptable. 
If the plan is deemed ready for full implementation after the pilot, the evaluator can assist by 
verifying to the stakeholders and leaders that the institutionalization is proceeding as planned and the 
extent to which the objectives of the institutionalization have been met. 

In all of this process, there are two keys: the technical knowledge and skills of the evaluator and 
the interaction of the evaluator with the stakeholders. The technical skills have to do with the quality 
of the evaluative components of the evaluation. The interaction has to do with changing the 
behaviors of the stakeholders, with moving the stakeholders to accept the changes that the new 
solution to this problem will bring. It’s about enabling the stakeholders to adapt, about the 
sociopolitical and/or psychological purpose of the evaluation. 

Moving to the Type III situation, both the problem and the solution are unclear to the 
stakeholders including the institutional leader. As with the unclear solution to the clear problem in 
Type II, that which is unclear requires more of the evaluator both in technical and interactive skills 
because the purpose is more than formative or summative. The evaluator has to assist the institutional 
leader to make the problem clear not only in the leader’s mind but also in the minds of the various 
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stakeholders. This begins with getting the stakeholders to delineate the current situation and then 
spotlighting information for the stakeholders in such ways as would enable them to view the current 
situation from several perspectives. 

Once there is some coming into focus about the current situation the evaluator can elicit 
characteristics of the desired situation. With these two aspects taking shape, each can be used to 
sharpen the other by helping the stakeholders work on contrasts between the current situation and its 
underlying values and the desired situation and its underlying values. When this gap evident and 
accepted by the stakeholders, the evaluator might be done with this phase. However, if the 
stakeholders simply accept that there is a gap and view it as insoluble or see only old problem 
resolution protocols as appropriate, the problem is not clear enough to resolve. In this situation the 
evaluator needs to assist the stakeholders to see the problem so clearly that they see both the need for 
new responses and the potential for responses as within the capacity of the institution. They need to 
develop an institutional sense both of need and of capacity to change. Among the tasks that the 
evaluator could use to build this clarity are having the stakeholders determine a means or criteria for 
measuring movement toward resolution, the knowledge and skill resources that would be necessary to 
resolve the problem, and the existence of any of that knowledge or skills within the institution or 
readily available from external sources. 

With the problem brought into focus and the solution and its implementation as yet unclear, the 
problem has become like a Type II problem. The evaluator has formative and/or summative 
evaluation work to do along with meeting the psychological and sociopolitical needs of the 
institution. As above, the evaluator would work to enable the stakeholders to understand and accept 
the causes and root causes of the problem, to develop criteria for acceptable resolution of the 
problem, to sort thorough potential solutions, to plan a process for monitoring the implementation of 
the selected solution, to help locate, monitor, and evaluate a pilot of the solution, and to develop 
information related to the institutionalization of the solution. 
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V 






In summary, evaluators have different roles to play within the institutional setting of their work 
and those roles depend on the institutional setting. Where there is clarity about both the problem and 
the solution to the problem, the work is within purposes that are summative and/or formative and 
hence basically technical. Where is there lack of clarity about the problem and/or the solution, both 
in terms of what it is and how to implement it, the work expands into sociopolitical and psychological 
purposes and becomes both technical and interactive. 

Heifetz, R. A. 1994. Leadership without easy answers. Cambridge, MA: Belknap Press. 

Nevo, D. 1990. “Role of the evaluator” in “Normative Dimensions of Evaluation Practice” in 
Walberg, H.J. and Haertel, G. D., The international encyclopedia of educational evaluation. New 
York: Pergamon Press. 
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